Modular Networks: Learning to Decompose Neural Computation

Scaling model capacity has been vital in the success of deep learning. For a typical network, necessary compute resources and training time grow dramatically with model size. Conditional computation is a promising way to increase the number of parameters with a relatively small increase in resources. We propose a training algorithm that flexibly chooses neural modules based on the data to be processed. Both the decomposition and modules are learned end-to-end. In contrast to existing approaches, training does not rely on regularization to enforce diversity in module use. We apply modular networks both to image recognition and language modeling tasks, where we achieve superior performance compared to several baselines. Introspection reveals that modules specialize in interpretable contexts.Comment: NIPS 201

Transfer Learning for Speech Recognition on a Budget

End-to-end training of automated speech recognition (ASR) systems requires massive data and compute resources. We explore transfer learning based on model adaptation as an approach for training ASR models under constrained GPU memory, throughput and training data. We conduct several systematic experiments adapting a Wav2Letter convolutional neural network originally trained for English ASR to the German language. We show that this technique allows faster training on consumer-grade resources while requiring less training data in order to achieve the same accuracy, thereby lowering the cost of training ASR models in other languages. Model introspection revealed that small adaptations to the network's weights were sufficient for good performance, especially for inner layers.Comment: Accepted for 2nd ACL Workshop on Representation Learning for NL

Epigenetic upregulation of FKBP5 by aging and stress contributes to NF-kappa B-driven inflammation and cardiovascular risk

Aging and psychosocial stress are associated with increased inflammation and disease risk, but the underlying molecular mechanisms are unclear. Because both aging and stress are also associated with lasting epigenetic changes, a plausible hypothesis is that stress along the lifespan could confer disease risk through epigenetic effects on molecules involved in inflammatory processes. Here, by combining large-scale analyses in human cohorts with experiments in cells, we report that FKBP5, a protein implicated in stress physiology, contributes to these relations. Across independent human cohorts (total n > 3,000), aging synergized with stress-related phenotypes, measured with childhood trauma and major depression questionnaires, to epigenetically up-regulate FKBP5 expression. These age/stress-related epigenetic effects were recapitulated in a cellular model of replicative senescence, whereby we exposed replicating human fibroblasts to stress (glucocorticoid) hormones. Unbiased genome-wide analyses in human blood linked higher FKBP5 mRNA with a proinflammatory profile and altered NF-kappa B-related gene networks. Accordingly, experiments in immune cells showed that higher FKBP5 promotes inflammation by strengthening the interactions of NF-kappa B regulatory kinases, whereas opposing FKBP5 either by genetic deletion (CRISPR/Cas9-mediated) or selective pharmacological inhibition prevented the effects on NF-kappa B. Further, the age/stress-related epigenetic signature enhanced FKBP5 response to NF-kappa B through a positive feedback loop and was present in individuals with a history of acute myocardial infarction, a disease state linked to peripheral inflammation. These findings suggest that aging/stress-driven FKBP5-NF-kappa B signaling mediates inflammation, potentially contributing to cardiovascular risk, and may thus point to novel biomarker and treatment possibilities.Peer reviewe

The Physics of the B Factories

This work is on the Physics of the B Factories. Part A of this book contains a brief description of the SLAC and KEK B Factories as well as their detectors, BaBar and Belle, and data taking related issues. Part B discusses tools and methods used by the experiments in order to obtain results. The results themselves can be found in Part C

Gaussian Mean Field Regularizes by Limiting Learned Information

Variational inference with a factorized Gaussian posterior estimate is a widely-used approach for learning parameters and hidden variables. Empirically, a regularizing effect can be observed that is poorly understood. In this work, we show how mean field inference improves generalization by limiting mutual information between learned parameters and the data through noise. We quantify a maximum capacity when the posterior variance is either fixed or learned and connect it to generalization error, even when the KL-divergence in the objective is scaled by a constant. Our experiments suggest that bounding information between parameters and data effectively regularizes neural networks on both supervised and unsupervised tasks

Französisch-deutscher Briefsteller und Haussecretär

deutscher Theil von Julius Kunze ; partie franc. par P. GauthierTexte jew. deutsch u. franz., nur dt. Text in Fraktu

Gaussian mean field regularizes by limiting learned information

Variational inference with a factorized Gaussian posterior estimate is a widely-used approach for learning parameters and hidden variables. Empirically, a regularizing effect can be observed that is poorly understood. In this work, we show how mean field inference improves generalization by limiting mutual information between learned parameters and the data through noise. We quantify a maximum capacity when the posterior variance is either fixed or learned and connect it to generalization error, even when the KL-divergence in the objective is scaled by a constant. Our experiments suggest that bounding information between parameters and data effectively regularizes neural networks on both supervised and unsupervised tasks